{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "\n",
    "Autograd: 自动求导机制\n",
    "PyTorch 中所有神经网络的核心是 autograd 包。 我们先简单介绍一下这个包，然后训练第一个简单的神经网络。\n",
    "autograd包为张量上的所有操作提供了自动求导。 它是一个在运行时定义的框架，这意味着反向传播是根据你的代码来确定如何运行，并且每次迭代可以是不同的。\n",
    "torch.Tensor是这个包的核心类。如果设置 .requires_grad 为 True，那么将会追踪所有对于该张量的操作。 当完成计算后通过调用 .backward()，自动计算所有的梯度， 这个张量的所有梯度将会自动积累到 .grad 属性。\n",
    "要阻止张量跟踪历史记录，可以调用.detach()方法将其与计算历史记录分离，并禁止跟踪它将来的计算记录。\n",
    "为了防止跟踪历史记录（和使用内存），可以将代码块包装在with torch.no_grad()：中。 在评估模型时特别有用，因为模型可能具有requires_grad = True的可训练参数，但是我们不需要梯度计算。\n",
    "在自动梯度计算中还有另外一个重要的类Function.\n",
    "Tensor 和 Function互相连接并生成一个非循环图，它表示和存储了完整的计算历史。 每个张量都有一个.grad_fn属性，这个属性引用了一个创建了Tensor的Function（除非这个张量是用户手动创建的，即，这个张量的 grad_fn 是 None）。\n",
    "如果需要计算导数，你可以在Tensor上调用.backward()。 如果Tensor是一个标量（即它包含一个元素数据）则不需要为backward()指定任何参数， 但是如果它有更多的元素，你需要指定一个gradient 参数来匹配张量的形状。\n",
    "在其他的文章中你可能会看到说将Tensor包裹到Variable中提供自动梯度计算，Variable 这个在0.41版中已经被标注为过期了，现在可以直接使用Tensor\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "import torch\n",
    "x = torch.ones(2,2,requires_grad=True)\n",
    "print(x)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([[1., 1.],\n",
      "        [1., 1.]], requires_grad=True)\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "source": [
    "y = x + 2\n",
    "print(y)\n",
    "print(y.grad_fn)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([[3., 3.],\n",
      "        [3., 3.]], grad_fn=<AddBackward0>)\n",
      "<AddBackward0 object at 0x7f5570054850>\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "source": [
    "z = y*y*3\n",
    "out = z.mean()\n",
    "print(\"z\", z)\n",
    "print(\"out \", out)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "z tensor([[27., 27.],\n",
      "        [27., 27.]], grad_fn=<MulBackward0>)\n",
      "out  tensor(27., grad_fn=<MeanBackward0>)\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "source": [
    "a = torch.randn(2,2)\n",
    "a=((a*3)/(a-1))\n",
    "print(a.requires_grad)\n",
    "a.requires_grad_(True)  #.requires_grad_( ... ) 可以改变现有张量的 requires_grad属性。 如果没有指定的话，默认输入的flag是 False。\n",
    "print(a.requires_grad)\n",
    "b =(a*a).sum()\n",
    "print(b.grad_fn)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "False\n",
      "True\n",
      "<SumBackward0 object at 0x7f5570023670>\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "out.backward()  #反向传播 因为 out是一个纯量（scalar），out.backward() 等于out.backward(torch.tensor(1))。\n",
    "print(x.grad)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([[4.5000, 4.5000],\n",
      "        [4.5000, 4.5000]])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "source": [
    "x  = torch.randn(3,requires_grad=True)\n",
    "y = x*2\n",
    "while y.data.norm() < 1000:\n",
    "    y = y*2\n",
    "print(y)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([-366.0309, -555.2794, -827.4849], grad_fn=<MulBackward0>)\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "在这个情形中，y不再是个标量。torch.autograd无法直接计算出完整的雅可比行列，但是如果我们只想要vector-Jacobian product，只需将向量作为参数传入backward："
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "source": [
    "gradients = torch.tensor([0.1,1.0,0.0001],dtype=torch.float)  #torch.tensor()仅仅是Python的函\n",
    "#torch.Tensor()是Python类，更明确的说，是默认张量类型torch.FloatTensor()的别名，torch.Tensor([1,2]) 会调用Tensor类的构造函数__init__，生成单精度浮点类型的张量。\n",
    "y.backward(gradients)\n",
    "print(x.grad)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([1.0240e+02, 1.0240e+03, 1.0240e-01])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "source": [
    "#如果.requires_grad=True但是你又不希望进行autograd的计算， 那么可以将变量包裹在 with torch.no_grad()中:\n",
    "print(x.requires_grad)\n",
    "print((x**2).requires_grad)\n",
    "with torch.no_grad():\n",
    "    print((x**2).requires_grad)"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "True\n",
      "True\n",
      "False\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "进阶\n",
    "要想通过Tensor类本身就支持了使用autograd功能，只需要设置.requires_grad=True\n",
    "\n",
    "Variable类中的的grad和grad_fn属性已经整合进入了Tensor类中\n",
    "\n",
    "\n",
    "Autograd 过程解析¶\n",
    "为了说明Pytorch的自动求导原理，我们来尝试分析一下PyTorch的源代码，虽然Pytorch的 Tensor和 TensorBase都是使用CPP来实现的，但是可以使用一些Python的一些方法查看这些对象在Python的属性和状态。 Python的 dir() 返回参数的属性、方法列表。z是一个Tensor变量，看看里面有哪些成员变量。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "source": [
    "import torch\n",
    "x = torch.rand(5,5,requires_grad=True)\n",
    "y = torch.rand(5,5,requires_grad=True)\n",
    "z = x**2 + y**3\n",
    "dir(z)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "['T',\n",
       " '__abs__',\n",
       " '__add__',\n",
       " '__and__',\n",
       " '__array__',\n",
       " '__array_priority__',\n",
       " '__array_wrap__',\n",
       " '__bool__',\n",
       " '__class__',\n",
       " '__complex__',\n",
       " '__contains__',\n",
       " '__deepcopy__',\n",
       " '__delattr__',\n",
       " '__delitem__',\n",
       " '__dict__',\n",
       " '__dir__',\n",
       " '__div__',\n",
       " '__doc__',\n",
       " '__eq__',\n",
       " '__float__',\n",
       " '__floordiv__',\n",
       " '__format__',\n",
       " '__ge__',\n",
       " '__getattribute__',\n",
       " '__getitem__',\n",
       " '__gt__',\n",
       " '__hash__',\n",
       " '__iadd__',\n",
       " '__iand__',\n",
       " '__idiv__',\n",
       " '__ifloordiv__',\n",
       " '__ilshift__',\n",
       " '__imod__',\n",
       " '__imul__',\n",
       " '__index__',\n",
       " '__init__',\n",
       " '__init_subclass__',\n",
       " '__int__',\n",
       " '__invert__',\n",
       " '__ior__',\n",
       " '__ipow__',\n",
       " '__irshift__',\n",
       " '__isub__',\n",
       " '__iter__',\n",
       " '__itruediv__',\n",
       " '__ixor__',\n",
       " '__le__',\n",
       " '__len__',\n",
       " '__long__',\n",
       " '__lshift__',\n",
       " '__lt__',\n",
       " '__matmul__',\n",
       " '__mod__',\n",
       " '__module__',\n",
       " '__mul__',\n",
       " '__ne__',\n",
       " '__neg__',\n",
       " '__new__',\n",
       " '__nonzero__',\n",
       " '__or__',\n",
       " '__pos__',\n",
       " '__pow__',\n",
       " '__radd__',\n",
       " '__rdiv__',\n",
       " '__reduce__',\n",
       " '__reduce_ex__',\n",
       " '__repr__',\n",
       " '__reversed__',\n",
       " '__rfloordiv__',\n",
       " '__rmul__',\n",
       " '__rpow__',\n",
       " '__rshift__',\n",
       " '__rsub__',\n",
       " '__rtruediv__',\n",
       " '__setattr__',\n",
       " '__setitem__',\n",
       " '__setstate__',\n",
       " '__sizeof__',\n",
       " '__str__',\n",
       " '__sub__',\n",
       " '__subclasshook__',\n",
       " '__torch_function__',\n",
       " '__truediv__',\n",
       " '__weakref__',\n",
       " '__xor__',\n",
       " '_backward_hooks',\n",
       " '_base',\n",
       " '_cdata',\n",
       " '_coalesced_',\n",
       " '_dimI',\n",
       " '_dimV',\n",
       " '_grad',\n",
       " '_grad_fn',\n",
       " '_indices',\n",
       " '_is_view',\n",
       " '_make_subclass',\n",
       " '_nnz',\n",
       " '_reduce_ex_internal',\n",
       " '_to_sparse_csr',\n",
       " '_update_names',\n",
       " '_values',\n",
       " '_version',\n",
       " 'abs',\n",
       " 'abs_',\n",
       " 'absolute',\n",
       " 'absolute_',\n",
       " 'acos',\n",
       " 'acos_',\n",
       " 'acosh',\n",
       " 'acosh_',\n",
       " 'add',\n",
       " 'add_',\n",
       " 'addbmm',\n",
       " 'addbmm_',\n",
       " 'addcdiv',\n",
       " 'addcdiv_',\n",
       " 'addcmul',\n",
       " 'addcmul_',\n",
       " 'addmm',\n",
       " 'addmm_',\n",
       " 'addmv',\n",
       " 'addmv_',\n",
       " 'addr',\n",
       " 'addr_',\n",
       " 'align_as',\n",
       " 'align_to',\n",
       " 'all',\n",
       " 'allclose',\n",
       " 'amax',\n",
       " 'amin',\n",
       " 'angle',\n",
       " 'any',\n",
       " 'apply_',\n",
       " 'arccos',\n",
       " 'arccos_',\n",
       " 'arccosh',\n",
       " 'arccosh_',\n",
       " 'arcsin',\n",
       " 'arcsin_',\n",
       " 'arcsinh',\n",
       " 'arcsinh_',\n",
       " 'arctan',\n",
       " 'arctan_',\n",
       " 'arctanh',\n",
       " 'arctanh_',\n",
       " 'argmax',\n",
       " 'argmin',\n",
       " 'argsort',\n",
       " 'as_strided',\n",
       " 'as_strided_',\n",
       " 'as_subclass',\n",
       " 'asin',\n",
       " 'asin_',\n",
       " 'asinh',\n",
       " 'asinh_',\n",
       " 'atan',\n",
       " 'atan2',\n",
       " 'atan2_',\n",
       " 'atan_',\n",
       " 'atanh',\n",
       " 'atanh_',\n",
       " 'backward',\n",
       " 'baddbmm',\n",
       " 'baddbmm_',\n",
       " 'bernoulli',\n",
       " 'bernoulli_',\n",
       " 'bfloat16',\n",
       " 'bincount',\n",
       " 'bitwise_and',\n",
       " 'bitwise_and_',\n",
       " 'bitwise_not',\n",
       " 'bitwise_not_',\n",
       " 'bitwise_or',\n",
       " 'bitwise_or_',\n",
       " 'bitwise_xor',\n",
       " 'bitwise_xor_',\n",
       " 'bmm',\n",
       " 'bool',\n",
       " 'broadcast_to',\n",
       " 'byte',\n",
       " 'cauchy_',\n",
       " 'cdouble',\n",
       " 'ceil',\n",
       " 'ceil_',\n",
       " 'cfloat',\n",
       " 'char',\n",
       " 'cholesky',\n",
       " 'cholesky_inverse',\n",
       " 'cholesky_solve',\n",
       " 'chunk',\n",
       " 'clamp',\n",
       " 'clamp_',\n",
       " 'clamp_max',\n",
       " 'clamp_max_',\n",
       " 'clamp_min',\n",
       " 'clamp_min_',\n",
       " 'clip',\n",
       " 'clip_',\n",
       " 'clone',\n",
       " 'coalesce',\n",
       " 'col_indices',\n",
       " 'conj',\n",
       " 'contiguous',\n",
       " 'copy_',\n",
       " 'copysign',\n",
       " 'copysign_',\n",
       " 'cos',\n",
       " 'cos_',\n",
       " 'cosh',\n",
       " 'cosh_',\n",
       " 'count_nonzero',\n",
       " 'cpu',\n",
       " 'cross',\n",
       " 'crow_indices',\n",
       " 'cuda',\n",
       " 'cummax',\n",
       " 'cummin',\n",
       " 'cumprod',\n",
       " 'cumprod_',\n",
       " 'cumsum',\n",
       " 'cumsum_',\n",
       " 'data',\n",
       " 'data_ptr',\n",
       " 'deg2rad',\n",
       " 'deg2rad_',\n",
       " 'dense_dim',\n",
       " 'dequantize',\n",
       " 'det',\n",
       " 'detach',\n",
       " 'detach_',\n",
       " 'device',\n",
       " 'diag',\n",
       " 'diag_embed',\n",
       " 'diagflat',\n",
       " 'diagonal',\n",
       " 'diff',\n",
       " 'digamma',\n",
       " 'digamma_',\n",
       " 'dim',\n",
       " 'dist',\n",
       " 'div',\n",
       " 'div_',\n",
       " 'divide',\n",
       " 'divide_',\n",
       " 'dot',\n",
       " 'double',\n",
       " 'dsplit',\n",
       " 'dtype',\n",
       " 'eig',\n",
       " 'element_size',\n",
       " 'eq',\n",
       " 'eq_',\n",
       " 'equal',\n",
       " 'erf',\n",
       " 'erf_',\n",
       " 'erfc',\n",
       " 'erfc_',\n",
       " 'erfinv',\n",
       " 'erfinv_',\n",
       " 'exp',\n",
       " 'exp2',\n",
       " 'exp2_',\n",
       " 'exp_',\n",
       " 'expand',\n",
       " 'expand_as',\n",
       " 'expm1',\n",
       " 'expm1_',\n",
       " 'exponential_',\n",
       " 'fill_',\n",
       " 'fill_diagonal_',\n",
       " 'fix',\n",
       " 'fix_',\n",
       " 'flatten',\n",
       " 'flip',\n",
       " 'fliplr',\n",
       " 'flipud',\n",
       " 'float',\n",
       " 'float_power',\n",
       " 'float_power_',\n",
       " 'floor',\n",
       " 'floor_',\n",
       " 'floor_divide',\n",
       " 'floor_divide_',\n",
       " 'fmax',\n",
       " 'fmin',\n",
       " 'fmod',\n",
       " 'fmod_',\n",
       " 'frac',\n",
       " 'frac_',\n",
       " 'frexp',\n",
       " 'gather',\n",
       " 'gcd',\n",
       " 'gcd_',\n",
       " 'ge',\n",
       " 'ge_',\n",
       " 'geometric_',\n",
       " 'geqrf',\n",
       " 'ger',\n",
       " 'get_device',\n",
       " 'grad',\n",
       " 'grad_fn',\n",
       " 'greater',\n",
       " 'greater_',\n",
       " 'greater_equal',\n",
       " 'greater_equal_',\n",
       " 'gt',\n",
       " 'gt_',\n",
       " 'half',\n",
       " 'hardshrink',\n",
       " 'has_names',\n",
       " 'heaviside',\n",
       " 'heaviside_',\n",
       " 'histc',\n",
       " 'hsplit',\n",
       " 'hypot',\n",
       " 'hypot_',\n",
       " 'i0',\n",
       " 'i0_',\n",
       " 'igamma',\n",
       " 'igamma_',\n",
       " 'igammac',\n",
       " 'igammac_',\n",
       " 'imag',\n",
       " 'index_add',\n",
       " 'index_add_',\n",
       " 'index_copy',\n",
       " 'index_copy_',\n",
       " 'index_fill',\n",
       " 'index_fill_',\n",
       " 'index_put',\n",
       " 'index_put_',\n",
       " 'index_select',\n",
       " 'indices',\n",
       " 'inner',\n",
       " 'int',\n",
       " 'int_repr',\n",
       " 'inverse',\n",
       " 'is_coalesced',\n",
       " 'is_complex',\n",
       " 'is_contiguous',\n",
       " 'is_cuda',\n",
       " 'is_distributed',\n",
       " 'is_floating_point',\n",
       " 'is_leaf',\n",
       " 'is_meta',\n",
       " 'is_mkldnn',\n",
       " 'is_mlc',\n",
       " 'is_nonzero',\n",
       " 'is_pinned',\n",
       " 'is_quantized',\n",
       " 'is_same_size',\n",
       " 'is_set_to',\n",
       " 'is_shared',\n",
       " 'is_signed',\n",
       " 'is_sparse',\n",
       " 'is_sparse_csr',\n",
       " 'is_vulkan',\n",
       " 'is_xpu',\n",
       " 'isclose',\n",
       " 'isfinite',\n",
       " 'isinf',\n",
       " 'isnan',\n",
       " 'isneginf',\n",
       " 'isposinf',\n",
       " 'isreal',\n",
       " 'istft',\n",
       " 'item',\n",
       " 'kron',\n",
       " 'kthvalue',\n",
       " 'layout',\n",
       " 'lcm',\n",
       " 'lcm_',\n",
       " 'ldexp',\n",
       " 'ldexp_',\n",
       " 'le',\n",
       " 'le_',\n",
       " 'lerp',\n",
       " 'lerp_',\n",
       " 'less',\n",
       " 'less_',\n",
       " 'less_equal',\n",
       " 'less_equal_',\n",
       " 'lgamma',\n",
       " 'lgamma_',\n",
       " 'log',\n",
       " 'log10',\n",
       " 'log10_',\n",
       " 'log1p',\n",
       " 'log1p_',\n",
       " 'log2',\n",
       " 'log2_',\n",
       " 'log_',\n",
       " 'log_normal_',\n",
       " 'log_softmax',\n",
       " 'logaddexp',\n",
       " 'logaddexp2',\n",
       " 'logcumsumexp',\n",
       " 'logdet',\n",
       " 'logical_and',\n",
       " 'logical_and_',\n",
       " 'logical_not',\n",
       " 'logical_not_',\n",
       " 'logical_or',\n",
       " 'logical_or_',\n",
       " 'logical_xor',\n",
       " 'logical_xor_',\n",
       " 'logit',\n",
       " 'logit_',\n",
       " 'logsumexp',\n",
       " 'long',\n",
       " 'lstsq',\n",
       " 'lt',\n",
       " 'lt_',\n",
       " 'lu',\n",
       " 'lu_solve',\n",
       " 'map2_',\n",
       " 'map_',\n",
       " 'masked_fill',\n",
       " 'masked_fill_',\n",
       " 'masked_scatter',\n",
       " 'masked_scatter_',\n",
       " 'masked_select',\n",
       " 'matmul',\n",
       " 'matrix_exp',\n",
       " 'matrix_power',\n",
       " 'max',\n",
       " 'maximum',\n",
       " 'mean',\n",
       " 'median',\n",
       " 'min',\n",
       " 'minimum',\n",
       " 'mm',\n",
       " 'mode',\n",
       " 'moveaxis',\n",
       " 'movedim',\n",
       " 'msort',\n",
       " 'mul',\n",
       " 'mul_',\n",
       " 'multinomial',\n",
       " 'multiply',\n",
       " 'multiply_',\n",
       " 'mv',\n",
       " 'mvlgamma',\n",
       " 'mvlgamma_',\n",
       " 'name',\n",
       " 'names',\n",
       " 'nan_to_num',\n",
       " 'nan_to_num_',\n",
       " 'nanmedian',\n",
       " 'nanquantile',\n",
       " 'nansum',\n",
       " 'narrow',\n",
       " 'narrow_copy',\n",
       " 'ndim',\n",
       " 'ndimension',\n",
       " 'ne',\n",
       " 'ne_',\n",
       " 'neg',\n",
       " 'neg_',\n",
       " 'negative',\n",
       " 'negative_',\n",
       " 'nelement',\n",
       " 'new',\n",
       " 'new_empty',\n",
       " 'new_empty_strided',\n",
       " 'new_full',\n",
       " 'new_ones',\n",
       " 'new_tensor',\n",
       " 'new_zeros',\n",
       " 'nextafter',\n",
       " 'nextafter_',\n",
       " 'nonzero',\n",
       " 'norm',\n",
       " 'normal_',\n",
       " 'not_equal',\n",
       " 'not_equal_',\n",
       " 'numel',\n",
       " 'numpy',\n",
       " 'orgqr',\n",
       " 'ormqr',\n",
       " 'outer',\n",
       " 'output_nr',\n",
       " 'permute',\n",
       " 'pin_memory',\n",
       " 'pinverse',\n",
       " 'polygamma',\n",
       " 'polygamma_',\n",
       " 'positive',\n",
       " 'pow',\n",
       " 'pow_',\n",
       " 'prelu',\n",
       " 'prod',\n",
       " 'put',\n",
       " 'put_',\n",
       " 'q_per_channel_axis',\n",
       " 'q_per_channel_scales',\n",
       " 'q_per_channel_zero_points',\n",
       " 'q_scale',\n",
       " 'q_zero_point',\n",
       " 'qr',\n",
       " 'qscheme',\n",
       " 'quantile',\n",
       " 'rad2deg',\n",
       " 'rad2deg_',\n",
       " 'random_',\n",
       " 'ravel',\n",
       " 'real',\n",
       " 'reciprocal',\n",
       " 'reciprocal_',\n",
       " 'record_stream',\n",
       " 'refine_names',\n",
       " 'register_hook',\n",
       " 'reinforce',\n",
       " 'relu',\n",
       " 'relu_',\n",
       " 'remainder',\n",
       " 'remainder_',\n",
       " 'rename',\n",
       " 'rename_',\n",
       " 'renorm',\n",
       " 'renorm_',\n",
       " 'repeat',\n",
       " 'repeat_interleave',\n",
       " 'requires_grad',\n",
       " 'requires_grad_',\n",
       " 'reshape',\n",
       " 'reshape_as',\n",
       " 'resize',\n",
       " 'resize_',\n",
       " 'resize_as',\n",
       " 'resize_as_',\n",
       " 'retain_grad',\n",
       " 'roll',\n",
       " 'rot90',\n",
       " 'round',\n",
       " 'round_',\n",
       " 'rsqrt',\n",
       " 'rsqrt_',\n",
       " 'scatter',\n",
       " 'scatter_',\n",
       " 'scatter_add',\n",
       " 'scatter_add_',\n",
       " 'select',\n",
       " 'set_',\n",
       " 'sgn',\n",
       " 'sgn_',\n",
       " 'shape',\n",
       " 'share_memory_',\n",
       " 'short',\n",
       " 'sigmoid',\n",
       " 'sigmoid_',\n",
       " 'sign',\n",
       " 'sign_',\n",
       " 'signbit',\n",
       " 'sin',\n",
       " 'sin_',\n",
       " 'sinc',\n",
       " 'sinc_',\n",
       " 'sinh',\n",
       " 'sinh_',\n",
       " 'size',\n",
       " 'slogdet',\n",
       " 'smm',\n",
       " 'softmax',\n",
       " 'solve',\n",
       " 'sort',\n",
       " 'sparse_dim',\n",
       " 'sparse_mask',\n",
       " 'sparse_resize_',\n",
       " 'sparse_resize_and_clear_',\n",
       " 'split',\n",
       " 'split_with_sizes',\n",
       " 'sqrt',\n",
       " 'sqrt_',\n",
       " 'square',\n",
       " 'square_',\n",
       " 'squeeze',\n",
       " 'squeeze_',\n",
       " 'sspaddmm',\n",
       " 'std',\n",
       " 'stft',\n",
       " 'storage',\n",
       " 'storage_offset',\n",
       " 'storage_type',\n",
       " 'stride',\n",
       " 'sub',\n",
       " 'sub_',\n",
       " 'subtract',\n",
       " 'subtract_',\n",
       " 'sum',\n",
       " 'sum_to_size',\n",
       " 'svd',\n",
       " 'swapaxes',\n",
       " 'swapaxes_',\n",
       " 'swapdims',\n",
       " 'swapdims_',\n",
       " 'symeig',\n",
       " 't',\n",
       " 't_',\n",
       " 'take',\n",
       " 'take_along_dim',\n",
       " 'tan',\n",
       " 'tan_',\n",
       " 'tanh',\n",
       " 'tanh_',\n",
       " 'tensor_split',\n",
       " 'tile',\n",
       " 'to',\n",
       " 'to_dense',\n",
       " 'to_mkldnn',\n",
       " 'to_sparse',\n",
       " 'tolist',\n",
       " 'topk',\n",
       " 'trace',\n",
       " 'transpose',\n",
       " 'transpose_',\n",
       " 'triangular_solve',\n",
       " 'tril',\n",
       " 'tril_',\n",
       " 'triu',\n",
       " 'triu_',\n",
       " 'true_divide',\n",
       " 'true_divide_',\n",
       " 'trunc',\n",
       " 'trunc_',\n",
       " 'type',\n",
       " 'type_as',\n",
       " 'unbind',\n",
       " 'unflatten',\n",
       " 'unfold',\n",
       " 'uniform_',\n",
       " 'unique',\n",
       " 'unique_consecutive',\n",
       " 'unsafe_chunk',\n",
       " 'unsafe_split',\n",
       " 'unsafe_split_with_sizes',\n",
       " 'unsqueeze',\n",
       " 'unsqueeze_',\n",
       " 'values',\n",
       " 'var',\n",
       " 'vdot',\n",
       " 'view',\n",
       " 'view_as',\n",
       " 'vsplit',\n",
       " 'where',\n",
       " 'xlogy',\n",
       " 'xlogy_',\n",
       " 'xpu',\n",
       " 'zero_']"
      ]
     },
     "metadata": {},
     "execution_count": 1
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "返回很多，我们直接排除掉一些Python中特殊方法（以__开头和结束的）和私有方法（以_开头的，直接看几个比较主要的属性： .is_leaf：记录是否是叶子节点。通过这个属性来确定这个变量的类型 在官方文档中所说的“graph leaves”，“leaf variables”，都是指像x，y这样的手动创建的、而非运算得到的变量，这些变量称为创建变量。 像z这样的，是通过计算后得到的结果称为结果变量。\n",
    "\n",
    "一个变量是创建变量还是结果变量是通过.is_leaf来获取的。\n",
    "\n"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "source": [
    "print(\"x.is_leaf= \"+ str(x.is_leaf))\n",
    "print(\"z.is_leaf= \"+ str(z.is_leaf))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "x.is_leaf= True\n",
      "z.is_leaf= False\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "\n",
    "x是手动创建的没有通过计算，所以他被认为是一个叶子节点也就是一个创建变量，而z是通过x与y的一系列计算得到的，所以不是叶子结点也就是结果变量。\n",
    "\n",
    "为什么我们执行z.backward()方法会更新x.grad和y.grad呢？ .grad_fn属性记录的就是这部分的操作，虽然.backward()方法也是CPP实现的，但是可以通过Python来进行简单的探索。\n",
    "\n",
    "grad_fn：记录并且编码了完整的计算历史"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "source": [
    "z.grad_fn\n",
    "#grad_fn是一个AddBackward0类型的变量 AddBackward0这个类也是用Cpp来写的，但是我们从名字里就能够大概知道，他是加法(ADD)的反反向传播（Backward），看看里面有些什么东西"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "<AddBackward0 at 0x7fa51c126a00>"
      ]
     },
     "metadata": {},
     "execution_count": 4
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "source": [
    "dir(z.grad_fn)\n",
    "z.grad_fn.next_functions\n",
    "#next_functions是一个tuple of tuple of PowBackward0 and int。\n",
    "#为什么是2个tuple ？ 因为我们的操作是z= x**2+y**3 刚才的AddBackward0是相加，而前面的操作是乘方 PowBackward0。tuple第一个元素就是x相关的操作记录\n",
    "xg = z.grad_fn.next_functions[0][0]\n",
    "dir(xg)"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "['__call__',\n",
       " '__class__',\n",
       " '__delattr__',\n",
       " '__dir__',\n",
       " '__doc__',\n",
       " '__eq__',\n",
       " '__format__',\n",
       " '__ge__',\n",
       " '__getattribute__',\n",
       " '__gt__',\n",
       " '__hash__',\n",
       " '__init__',\n",
       " '__init_subclass__',\n",
       " '__le__',\n",
       " '__lt__',\n",
       " '__ne__',\n",
       " '__new__',\n",
       " '__reduce__',\n",
       " '__reduce_ex__',\n",
       " '__repr__',\n",
       " '__setattr__',\n",
       " '__sizeof__',\n",
       " '__str__',\n",
       " '__subclasshook__',\n",
       " '_register_hook_dict',\n",
       " '_saved_exponent',\n",
       " '_saved_self',\n",
       " 'metadata',\n",
       " 'name',\n",
       " 'next_functions',\n",
       " 'register_hook',\n",
       " 'requires_grad']"
      ]
     },
     "metadata": {},
     "execution_count": 8
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "source": [
    "x_leaf =xg.next_functions[0][0]\n",
    "type(x_leaf)\n",
    "#在PyTorch的反向图计算中，AccumulateGrad类型代表的就是叶子节点类型，也就是计算图终止节点。AccumulateGrad类中有一个.variable属性指向叶子节点。"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "AccumulateGrad"
      ]
     },
     "metadata": {},
     "execution_count": 9
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "source": [
    "x_leaf.variable"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[0.1984, 0.8660, 0.5423, 0.1056, 0.7492],\n",
       "        [0.8248, 0.3125, 0.7376, 0.5506, 0.9036],\n",
       "        [0.4060, 0.5647, 0.7465, 0.8046, 0.0898],\n",
       "        [0.6959, 0.5429, 0.1786, 0.1837, 0.6830],\n",
       "        [0.1164, 0.7483, 0.4170, 0.0340, 0.2897]], requires_grad=True)"
      ]
     },
     "metadata": {},
     "execution_count": 10
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "source": [
    "print(\"x_leaf.variable的id: \"+str(id(x_leaf.variable)))\n",
    "print(\"x的id： \"+str(id(x))) #这个.variable的属性就是我们的生成的变量x"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "x_leaf.variable的id: 140346490174784\n",
      "x的id： 140346490174784\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "\n",
    "这样整个规程就很清晰了：\n",
    "\n",
    "当我们执行z.backward()的时候。这个操作将调用z里面的grad_fn这个属性，执行求导的操作。\n",
    "这个操作将遍历grad_fn的next_functions，然后分别取出里面的Function（AccumulateGrad），执行求导操作。这部分是一个递归的过程直到最后类型为叶子节点。\n",
    "计算出结果以后，将结果保存到他们对应的variable 这个变量所引用的对象（x和y）的 grad这个属性里面。\n",
    "求导结束。所有的叶节点的grad变量都得到了相应的更新\n",
    "最终当我们执行完c.backward()之后，a和b里面的grad值就得到了更新。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "\n",
    "扩展Autograd¶\n",
    "如果需要自定义autograd扩展新的功能，就需要扩展Function类。因为Function使用autograd来计算结果和梯度，并对操作历史进行编码。 在Function类中最主要的方法就是forward()和backward()他们分别代表了前向传播和反向传播。\n",
    "\n",
    "一个自定义的Function需要一下三个方法：\n",
    "\n",
    "__init__ (optional)：如果这个操作需要额外的参数则需要定义这个Function的构造函数，不需要的话可以忽略。\n",
    "\n",
    "forward()：执行前向传播的计算代码\n",
    "\n",
    "backward()：反向传播时梯度计算的代码。 参数的个数和forward返回值的个数一样，每个参数代表传回到此操作的梯度。"
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "source": [
    "from torch.autograd.function import Function"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "source": [
    "class MulConstant(Function):\n",
    "    @staticmethod\n",
    "    def forward(ctx, tensor, constant):  ## ctx 用来保存信息这里类似self，并且ctx的属性可以在backward中调用\n",
    "        ctx.constant = constant\n",
    "        return tensor*constant\n",
    "    @staticmethod\n",
    "    def backward(ctx, grad_output):\n",
    "        # 返回的参数要与输入的参数一样.\n",
    "        # 第一个输入为3x3的张量，第二个为一个常数\n",
    "        # 常数的梯度必须是 None.\n",
    "        return grad_output,None"
   ],
   "outputs": [],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "source": [
    "a = torch.rand(3,3,requires_grad=True)\n",
    "b=MulConstant.apply(a,5)\n",
    "print(\"a: \"+str(a))\n",
    "print(\"b: \"+str(b))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "a: tensor([[0.9060, 0.9078, 0.7849],\n",
      "        [0.2803, 0.8344, 0.9015],\n",
      "        [0.9051, 0.8383, 0.2805]], requires_grad=True)\n",
      "b: tensor([[4.5302, 4.5392, 3.9246],\n",
      "        [1.4016, 4.1721, 4.5074],\n",
      "        [4.5255, 4.1914, 1.4024]], grad_fn=<MulConstantBackward>)\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "source": [
    "print(torch.ones_like(a))"
   ],
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "tensor([[1., 1., 1.],\n",
      "        [1., 1., 1.],\n",
      "        [1., 1., 1.]])\n"
     ]
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "source": [
    "b.backward(torch.ones_like(a))\n",
    "a.grad"
   ],
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "tensor([[1., 1., 1.],\n",
       "        [1., 1., 1.],\n",
       "        [1., 1., 1.]])"
      ]
     },
     "metadata": {},
     "execution_count": 19
    }
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "source": [],
   "outputs": [],
   "metadata": {}
  }
 ],
 "metadata": {
  "orig_nbformat": 4,
  "language_info": {
   "name": "python",
   "version": "3.8.10",
   "mimetype": "text/x-python",
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "pygments_lexer": "ipython3",
   "nbconvert_exporter": "python",
   "file_extension": ".py"
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.8.10 64-bit ('d2l-zh': conda)"
  },
  "interpreter": {
   "hash": "0bedf3452ed48e38a2b75ff9f66ce8ae9ae2e92e8665cb618ee0797d2f46b9e5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}